Method and device for correcting distortion errors due to accommodation effect in stereoscopic display

ABSTRACT

The invention concerns a method and a device for correcting distortion errors in a 3D content viewed by an observer on a screen. The method comprises the step of determining a rendered roundness factor (ITO of a pinhole model projected cylinder, estimating a rendered roundness factor table depending on the defined distance of the observer to the screen and the disparity values of objects of the image, determining for the observer a disparity transform function (TD) as a function of the estimated rendered roundness factor table and modifying the object disparity values using the disparity transform so that a perceived roundness factor of one is provided.

FIELD OF THE INVENTION

The invention concerns a method and a device for correcting distortionerrors due to accommodation effect in stereoscopic display. It takesplace in the domain of 3D rendering. In the present invention, thedisplayed or processed video content is either a stereoscopic ormulti-view 3D content.

TECHNOLOGICAL BACKGROUND OF THE INVENTION

A video content in stereoscopic 3D can be seen if a single image isshown for each eye. The differences between both images are called“disparities” which are the slight differences in the points of view.

Given the manner stereoscopic video content are displayed, spectatorshave to decouple both physiological systems called accommodation andconvergence. As a matter of fact, on-screen disparities stimulate theconvergence system while the eye accommodation of observer must keep astate to preserve the depth of field around the screen plane in orderfor vision to remain sharp.

In order to see an object in stereoscopy 3D, spectators have todetermine an observation distance on which disparities will be scaled.Thus, observers of stereoscopic content can take into account at leasttwo pieces of information from the visual sequence so as to perceive thesimulated depth: convergence which is associated to the disparity signaland accommodation which is associated to the spatial frequencies on thescreen. However, the signal of accommodation tends to be predominant inthe elaboration of the scaling distance of binocular disparity. Thisleads to a perceived depth different from the proposed depth.

It corresponds to an over-estimation of depth for crossed disparities(depth in front of the screen) and under-estimation for uncrosseddisparities (depth behind the screen). This phenomenon is furtherdescribed in the document by Watt et al. (2005). Focus cues affectperceived depth. Journal of Vision, 5(10):7, 834-862.

This phenomenon associated to accommodation directly impacts the qualityof 3D experience since the 3D space is perceived as distorted. Thiseffect has also been described as corresponding to the “roundnessfactor” which corresponds to the object width Dx divided by the objectdepth Dz for a round object. In FIG. 3, for example, the “roundnessfactor” of the represented element having a negative crossed disparityis 2. It measures how much the object proportions are affected by thedistortion. A perfect roundness factor of 1 indicates that no distortionis perceived.

The graph of FIG. 4 displays for a given observer the estimation of thedepth Dz of stereoscopic cylinders relative to their width Dx andsimulated distance Dvirtual of the observer from the screen plane. Xaxis represents the stereoscopic (simulated) distance of the cylinder tothe observer, Y left axis represents the observer estimation of thecylinder depth and Y right Axis represents the ratio estimated objectdepth/actual object width. The graph also shows two linear regressions:the first (1) is the regression line for dots representing theestimation of the cylinder depth, the second (2) is for crossesrepresenting the ratio estimated depth on actual width of the cylinder.These regression lines can be used for adjusting stereoscopic content aslong as they provide the transform which is operated by the visualsystem of the observer. The width of the cylinder is always equal to 70pixels. The magnitude of disparity required to perceive each cylinderincreases with the simulated distance. The depth of the cylinder istherefore more and more over-estimated when the cylinder approaches theobserver (in front of the screen) and under-estimated when the objectgoes away (behind the screen). Thus, a distortion of the cylinderappears.

The correction of this distortion is not solved currently. There is noproposition to provide a correct stereoscopic perception.

SUMMARY OF THE INVENTION

The invention proposes to remedy this problem. The invention concerns amethod for correcting distortion errors in a 3D content viewed by anobserver on a screen.

The method comprises the step of determining a rendered roundness factorof a pinhole model projected cylinder estimating a rendered roundnessfactor table depending on the defined distance of the observer to thescreen and the disparity values of objects of the image, determining forthe observer a disparity transform function as a function of theestimated rendered roundness factor table and modifying the objectdisparity values using the disparity transform so that a perceivedroundness factor of one is provided.

This solution allows for a fidelity restitution of the 3D content.

According to an embodiment of the invention the roundness factor of anobject corresponds to the object width divided by the object depth for around object.

According to an embodiment of the invention the disparity values areextraxted from a disparity map associated to the 3D content.

According to an embodiment of the invention the disparity values arecalculated from parameters of the observer associated to the 3D content.

According to an embodiment of the invention the disparity and distanceto the screen of an object of the 3D content is defined as the averageof disparity and average of distance to the screen of each pixel of theobject.

According to an embodiment of the invention the object disparity valuesis modified according to the original values so that the perceivedroundness factor equal to unity (prf=1.0) for the observer.

According to an embodiment of the invention the parameters to modify thedisparity values are directly implemented in 3D creation software.

The invention proposes also a device for correcting distortion errors ina 3D content viewed by an observer on a screen, The device comprisesmeans for determining a rendered roundness factor of a pinhole modelprojected cylinder, means for estimating a rendered roundness factortable depending on the defined distance of the observer to the screenand the disparity values of objects of the image, means for determiningfor the observer a disparity transform function as a function of theestimated rendered roundness factor table and means for modifying theobject disparity values using the disparity transform so that aperceived roundness factor of one is provided.

According to an embodiment of the invention, the means for determining arendered roundness factor corresponds to the cylinder width divided bythe cylinder depth for a round object.

According to an embodiment of the invention the disparity values areextracted from a disparity map associated to the 3D content byextracting means.

According to an embodiment of the invention, the disparity values arecalculated from parameters of the observer associated to the 3D contentby calculating means.

According to an embodiment of the invention, the disparity and distanceto the screen of an object of the 3D content is defined as the averageof disparity and average of distance to the screen of each pixel of theobject.

According to an embodiment of the invention, the means for modifying theobject disparity values modifies the object disparity values accordingto the original values so that the perceived roundness factor equal tounity for the observer.

According to an embodiment of the invention, the means for modifying theobject disparity values are directly implemented in means for capturingthe real world or in means for capturing the 3D drawing of 3D creationsoftware.

BRIEF DESCRIPTION OF THE FIGURES

The above-mentioned and other features and advantages of this invention,and the manner of attaining them, will become more apparent and theinvention will be better understood by reference to the followingdescription of embodiments of the invention taken in conjunction withthe accompanying drawings, wherein:

FIG. 1 is an illustration of the perspective projection predict for eacheye;

FIG. 2 shows a representation of the disparity computed from the pinholecamera model;

FIG. 3 illustrates the geometrical representation of a cylinder beyond ascreen for an observer;

FIG. 4 shows a graph representing the object disparity (left Y-axis) orthe roundness factor (right Y-axis) relative to the distance of thecylinder to the observer necessary to perceive undistorted object;

FIG. 5 is a schematic representation of a method according to apreferred embodiment of the invention.

DETAILED DESCRIPTION OF AN EMBODIMENT

The exemplifications set out herein illustrate preferred embodiments ofthe invention, and such exemplifications are not to be construed aslimiting the scope of the invention in any manner.

A method to modify the disparities of the 3D content as a function oftheir magnitude and their sign is described so that each variation ofdisparity induces a constant variation of depth. The sign of thedisparity is positive for an object represented behind the screen andnegative for an object represented before the screen. This methodenables the correction of stereoscopic distortions from the pinholecamera model so that correct stereoscopic perception to localize 3Dobject at the good position in depth are provided. These distortionscorrespond to the perception of elongated or flattened 3D shapes. Thepinhole camera model is prevalent 3D content creation model widely usedfor both synthetic and natural content, either as a GCI camera model, oras a representation model for real cameras. It is used as well as basisfor 3D image processing of many sorts.

As represented by the FIG. 1 perspective projection predicts for eacheye:

$\frac{x}{d} = \frac{X}{D}$ $\frac{y}{d} = \frac{Y}{D}$

With this model the disparity Δx between views for a point to be shownat distance D and for a display at distance d of a viewer is predictedto be

${\Delta \; x} = \frac{e \cdot d}{D}$

where e is the interocular or inter-pupillary distance of the viewer(see Error! Reference source not found.).

Today this basic pinhole camera model (stenope camera) is widely used todesign cameras, in 3D computed generated imagery (CGI) or for 3D imagesor video processing. This generates 3D images and video with scale/depthdistortion or incorrect 3D perspective and motion distortion whencompared to human vision.

Compared to natural viewing the pinhole camera model creates images moredifficult to visualize, although they are accepted as 3D images by thehuman visual system. The correspondence with natural space is onlypartial, valid only for small visual angles around the attention point.

A new proposed solution also consists in modifying the 3D content inorder to provide a correct 3D rendering in space since the pinhole modeldoes not take into account this distortion. The pinhole model is acentral projection model which only takes into account the geometry ofthe 3D space independently of the optics and physiology of the eye.

FIG. 3 illustrates the geometrical representation of a cylinder beyond ascreen for an observer. With crossed disparities, the cylinder isperceived elongated because disparities are overestimated. This is whathappens when the rendered object (dashed blue circle) does not take intoaccount the described distortion.

Considering that when the distance of the observer relative to thescreen plane D_(screen) is known, it is possible to retrieve and storehow an observer will perceive the roundness of a given stereoscopiccylinder of radius r with a given disparity.

Thus a roundness factor table or function is estimated depending on theobserver distance to the screen D_(screen) and presented disparities foran observer. This table or function can be inverted to obtain thedisparities providing a roundness factor of 1 for a given objectdistance to the screen.

The solution proposed by the invention is thus to re-compute thedisparities of the 3D content to provide a veridical correct perceptionof depth corresponding to a roundness factor equal to one.

Consequently, captured physical objects or CGI rendered objects will beperceived as the correct depth and with a correct shape whenstereoscopically displayed.

Different alternatives to this solution are part of the invention.

An object created with an initial projection model (e.g. the pinholeprojection model) is presented to an observer on a stereoscopy displaywith disparity and position on screen. This disparity and position onscreen is defined as the average of the disparity and distance to thescreen of each part of this object in the visual space of the givenobserver.

Per example, a local volume is given to an object, for example acylinder, with points in disparity computed relative to each observer'seye position, and depending on the average distance to the screen. Therendered roundness of this object, defined by relative disparities, is avariable in order to present to the observer different instances of theobject with different rendered roundness factors. The rendered roundnessfactor

rrf=Dz/Dx

where Dz is the depth measure and Dx is the width measure of a 3D roundobject [e.g. a flattened or elongated cylinder] projected with theinitial projection model. The rendered roundness factor rrfcharacterizes the intended flatness or elongation of the renderedpseudo-cylinder.

In varying the average distance to the screen, D_(screen), of theobserver, and in function of the rendered roundness factor (rrf), aperceived roundness factor (prf) is estimated for each observer. Theresults are stocked in a table t so that:

prf=t(D _(screen) , rrf),

which can be interpolated to a function f: prf=f (D_(screen), rrf).

A solution consists to solve the interpolated function f (D_(screen),rrf)=1.0 in order to find the distance and rendered roundness factor(dist, rrf) parameters of configurations preserving the perceivedroundness factor to one (prf=1.0).

An another solution consists in first deriving the function

rrf _((prf=1)) =g(dist)

giving the rendering roundness factor for each distance to the screenD_(screen) which will create a perceived roundness factor equal to unity(prf=1.0) for the observer and second in modifying the disparities in animage or video, originally created with the initial projection model,with the function rrf_((prf=1))=g(dist).

Obtention of the stereoscopic content can be obtained either in a firstcase through a visual capture of the real world or in a second case froma capture of 3D drawing.

In the first case, as the stereoscopic content is obtained through avisual capture of the real world, an algorithm modifies the magnitudesof the disparity according to the original values so that the perceivedroundness factor equal to unity (prf=1.0) for the observer.

In the second case, as the stereoscopic content is obtained from acapture of 3D drawing, the parameters to modify the disparity amplitudesare directly implemented in the 3D creation software such as a CGI (forCompute Generated Imagery).

Some particularities are related to inter-individual variability such asthe slope of the linear regression depicted on the FIG. 4. It can likelyvary depending of spectators. Thus, the content has to be adjustedaccording to their characteristics.

The weighting model between accommodation and vergence can be used asdeterminator of correcting factor, instead of the punctual estimation ofobserver characteristics. It merely assigns one weight to each cue. Thecorresponding weight can be derived from the literature and averageddata from observers' estimations.

As example and according to the classical model, the disparity of thecenter of a cylinder:

disparity_(c) =IOD−IOD·D _(screen) /D _(virtual),

IOD is the inter-ocular distance, D_(screen) is the distance from theobserver to the screen and D_(virtual) is the simulated distance whichone wants to display for a viewing distance to the screen Dscreen.

According to FIG. 3, an object located at D_(virtual)=1.1 m relative tothe observer will have an on-screen disparity of disp=12 mm for itsreference point for a screen distance of D_(screen)=1.3 m. The centermatches with a point f of fixation corresponding to the distance at:

D _(fixation) =IOD·D _(screen)/(IOD−disp).

The nearer point belonging to the cylinder around the center of theobject will have a disparity of:

disp=IOD−IOD·D _(screen)/(D _(fixation)−2r)

r being the radius of the cylinder. The value of the <<roundnessfactor>> is rrf=Dz/Dx for the perceived-pinhole model projected-cylinderat distance D_(fixation).

According to this present correction, based on observer data, thecorrected disparity which should be displayed to give rise to theperception of a “perfect cylinder” is:

disp_(corrected) =IOD−IOD·D _(screen)(D _(fixation)−2r/rrf).

An object appearing flattened in z with the pinhole projection model(e.g. rrf=0.5) has to be presented extended in z in a correctedprojection image pair to be perceived with the adequate/intended depth(e.g. r becomes r/0.5=2·r).

This correction may be realized for each point of observation on visualsequence.

When considering objects of local depth Δz (Δz would be equal to 2r forthe above cylinders) the true disparity transform is depending on thelocal depth of the object Δz and of the perceived roundness factor rrfwith the pinhole model. It is of the form:

disp_(true)=disp_(corrected) +IOD·D _(screen)(1/(D _(fixation) −Δz)−1/(D_(fixation)−2r/rrf))

according to the T_(D) transform of FIG. 4.

One mode of implementation consists in a content adaptation as shown bythe FIG. 5.

It comprises:

-   -   A first step consisting in a temporal segmentation of the        content and an estimation of the disparity map. The temporal        segmentation permits to separate a unique image for which a        disparity map will be extracted.    -   Then, a second step consists in an object analysis phase. A        content analysis and object segmentation permits to determine a        reference point of objects in the content. For this        determination, a saliency map based on psychological attention        models can be used. A particular gaze point corresponding to an        important saliency which capt the attention of the observer is        also determined in the image.    -   A third step consists in an estimation of the object disparity        D_(object) for each frame in a sequence. Binocular eye-gaze        tracking can be used to estimate the points-of-gaze (POG) of a        subject in real-world three-dimensional (3D) space using the        vergence of the eyes.    -   Next step consists in the establishment of the disparity        transform T_(D) taking account of observer characteristic such        as interocular distance and distance to the screen and    -   In a next step the modification of the disparity values of the        content is done through the transform T_(D).

Potential applications concern all stereoscopic content such as movies,TV, games, medical imaging and may be calibrated as a function of usercharacteristics. This solution would be also useful for applicationsrequiring high precision in stereoscopic imaging.

1. Method for correcting distortion errors in a 3D content viewed by an observer on a screen; the method being wherein it comprises the step of determining a rendered roundness factor (rrf) for the observer of the 3D content depending of the defined distance of the observer to the screen and of the disparity; determining for the observer a disparity transform function (TD) as a function of the determined rendered roundness factor so that the disparity values of objects of the 3D content are corrected for obtaining a perceived roundness factor of one.
 2. Method for correcting distortion errors in a 3D content as claimed in claim 1 wherein the parameters to modify the disparity values are directly implemented in a 3D creation software.
 3. Method for correcting distortion errors in a 3D content as claimed in claim 1 wherein the rendered roundness factor corresponds to the object width divided by the object depth for a round object.
 4. Method for correcting distortion errors in a 3D content as claimed in claim 1 wherein the disparity values are extracted from a disparity map associated to the 3D content.
 5. Method for correcting distortion errors in a 3D content as claimed in claim 1 wherein the disparity values are calculated from parameters of the observer associated to the 3D content.
 6. Method for correcting distortion errors in a 3D content as claimed in claim 4 wherein the disparity and distance to the screen of an object of the 3D content is defined as the average of disparity and average of distance to the screen of each pixel of the object.
 7. Device for correcting distortion errors in a 3D content viewed by an observer on a screen, wherein it comprises means for determining a rendered roundness factor (rrf) for the observer of the 3D content depending of the defined distance of the observer to the screen and of the disparity; means for determining for the observer a disparity transform function (TD) as a function of the determined rendered roundness factor so that the disparity values of objects of the 3D content are corrected for obtaining a perceived roundness factor of one.
 8. Device for correcting distortion errors in a 3D content as claimed in claim 7 wherein the means for determining a rendered roundness factor (rrf) divides the cylinder width by the cylinder depth for a round object.
 9. Device for correcting distortion errors in a 3D content as claimed in claim 7 wherein the disparity values are extracted from a disparity map associated to the 3D content by extracting means.
 10. Device for correcting distortion errors in a 3D content as claimed in claim 7 wherein the disparity values are calculated from parameters of the observer associated to the 3D content by calculating means.
 11. Device for correcting distortion errors in a 3D content as claimed in claim 11 wherein the disparity and distance to the screen of an object of the 3D content is defined as the average of disparity and average of distance to the screen of each pixel of the object.
 12. Device for correcting distortion errors in a 3D content as claimed in claim 8 wherein the means for modifying the object disparity values are directly implemented in means for capturing the real world or in means for capturing the 3D drawing of a 3D creation software. 